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Abstract 


The authors viewed participatory management and evaluation approaches as vehicles 
internal evaluators may use to effectively facilitate program evaluation within the new paradigm 
defined by the proliferating site-based management form of educational governance. They 
demonstrated that even though the new paradigm mandates the evaluation and decision-making 
roles to be within the purview of the school site, internal evaluators can assume effective roles 
which facilitate school decision-making bodies in approaching professional standards. They 
show that these roles can help decision-makers make quality evaluative judgements as well as 
empower stakeholders with meaninful input and opportunities to learn about disciplined inquiry. 
They reviewed their experiences by describing and critiquing three recent participatory 
evaluation projects in their district. The critiques were based on a checklist derived from extant 
professional standards and participatory evaluation literature. 


Lessons learned from their recent participatory experiences included the following. Two 
things should occur early in the project. First, stakeholders should be engaged in defining or 
redefining the goals and in the conduct of the project. Second, participatory evaluators should 
estabiish the facilitative nature of their role with the stakeholders but should be prepared and 
willing to assume various reponsibilities as needs arise. When thé evaluator is truly facilitative, 
stakeholders will help nurture trusting relationships which lead to quality evaluation and will 
assume responsibility for the project. By taking ownership, stakeholders seem to acquire a 
propensity to utilize the products, and decision-makers are encouraged to take consequential 
actions. Furthermore, when actions occur which are consequences of participatory evaluation, 
a propensity for follow-through evaluation among stakeholders seems to arise naturally. 


Participatory Approaches to Evaluation for Supporting School 
Management: Three Case Studies 


Debates over school reform, decentralization, and school governance have been ocurrinj, 
in many states. Site-based management, one of today’s most popular forms of decentralized 
governance, involves shifting decision-making from school boards, superintendents and central 
offices to individual schools and increasing stakeholder involvement. Sie-based management 
is, in fact, a reform of the entire school system. Schools cannot change their modes of 
operation if the central office does not change. School districts and their evaluation units need 
to learn how to help schools become strong, competent organizations so that they can use their 
independence to improve students’ education. 


Through management and evaluation activities, stakeholders begin to empower themselves 
and become responsible for the decisions affecting the school. This new paradigm impacts the 
way school districts interact with school staff and community to conduct planning and evaluation 
projects. Rather than control and regulate schools, central office staffs need to learn how to 
collaborate and involve stakeholders as partners in program management and evaluation. Central 
to this process is the belief that the district staff’s role in a decentralized school system is to 
create learning systems that enhance school improvement. 


Florida has adopted this new: paradigm. Under the legislative mandate of Florida 
Blueprint 2000: A System of School Improvement and Accountability, Florida schools must 
reorganize before the year 2000 by involving and connecting stakeholders in improvement 
processes. Every public school must have a School Advisory Council comprised of teachers, 
students, parents, and members of the community. The group is responsible for setting goals, 
developing and monitoring implementation plans, and conducting evaluations of school 
improvement. This milieu presents district evaluation units with a challenge to find roles which 
facilitate each school’s efforts in effectively carrying out the legislative mandate within the limits 
of its potential resources. The authors’ district evaluation unit is responding by experimenting 
with participatory management and evaluation approaches. The unit’s ultimate goal is to 
implement approaches which fit the mandate and the expanding demand for evaluation as well 
as to help site decision-makers understand and incorporate professional standards in their newly 
acquired evaluation roles. 


The purposes of this paper are to (a) introduce a checklist (see Appendix A) developed 
by the authors for critiquing participatory planning and evaluation projects and (b) to describe 
the participatory approaches used in the three Leon County Schools projects listed below. A 
rationale for the contents of the checklist is provided in the first section of this paper. The 
second section is devoted to applying the checklist to the three projects and evaluating them in 
respect to their foci, stakeholder identification/involvement, organization and negotiation 
processes, technical methods, reporting, and evaluation utilization/benefits. The paper is 


concluded with some issues and lessons learned by the authors from their recent experiences with 
participatory approaches. 


Leon County School District Projects 


A High Stakes Participatory Evaluation of an Innovative School Program, (1993-94) 
This evaluation was mandated by the site decision-making council after being initiated 
by the superintendent. (The future of the school’s alternative education program was at 
stake.) The evaluation unit responded to the school’s request for help by introducing a 
participatory approach focussd on producing quality evaluation transactions in a polarized 
atmosphere. Increasing the evatuative capabilities of the school was a seconcary 
concern. 


Conflict Mediation Tracking A Self-Evaluation Model for School P 
(1994) This grant project was managed by the evaluation unit using a participatory 
approach. A study group composed of school conflict mediation coordinators and expert 
consultants was formed to devise a data-driven management system which would have 
utility for school improvement planning and increase schools’ evaluative capabilities. 
Facilitated by the unit, the group determined the objectives and disposition of work. 


Technological Initiatives Guide: Leon C Pilot Schools Share Their Experi 
(1994 winner of the AERA Division H competition for innovative reporting) The 
evaluation unit conducted the project under the mandate of the district. The unit recruited 
pilot schools to participate in evaluating and sharing their technological practices and 
knowledge. 


RATIONALE FOR DEVELOPING A PARTICIPATORY EVALUATION CHECKLIST 


The checklist developed by the authors to guide planning, development, and critique of 
projects is provided in Appendix A. It contains appropriate evaluative questions for each of six 
areas corresponding to the following steps in an evaluation process. 


Focusing the evaluation 

Identifying Stakeholders 

Organization and Negotiation Process 
Methodology 
Reporting 

Utilization and Benefits 


The following is a discussion of some of the rationale for conducting participatory 
approaches in evaluation. It is arranged according to the six evaluation steps above which define 
the areas of the authors’ checklist. 


Focusing the Evaluation: Definition of Participatory Evaluation 

Garaway (1995) suggested that participatory activity in evaluation has a wide range of 
meanings. Making an analogy with action research, she maintained that the activity can be as 
simple as answering a survey or developing an instrument, or as extensive as complete 
involvement in the investigative process. She suggested that participatory evaluation has its 
foundation in the action research process. She contended that the participatory research process 
(PAR) has been widely used in the developing woild, especially in non-formal education projects 
- such as helping local farmers assess and solve problems, Whyte (1991) has defined PAR as a 
form of applied research, where the researcher becomes a facilitator in helping those being 
studied to also become actively engaged in the research process. 


Cousins & Earl (1992, p.399) defined participatory evaluation as, “applied social research 
that involves a partnership between trained evaluation personnel and practice-based decision 
makers, organization members with program responsibility, or people with a vital interest in the 
program." 


Easton (1993), defined participation as " a process by which individuals (or groups of 
people) assume a new level of responsibility for decision-making and action about goals, 
methods and resources in some realm of organized social activity of immediate concern to them, 
and through which they acquire or produce the new knowledge, skills and attitudes needed to 
play these enhanced roles." 


Brunner & Guzman (1989) defined participatory evaluation as an education process that 
produces action-oriented knowledge. They described the role of the evaluator in this process as 
“a methodological consultant" who is responsible for organizing and implementing the evaluation. 


Stakeholder Identification 

Participatory evaluation has also been identified with the stakeholder approach developed 
in the late 1970s by the National Institute of Education. Bryk (1983, p.1) defined the 
stakeholder approach as “an attempt to reorder the evaluator’s relationship with the groups 
involved in the program—that is, the people whose lives are affected by the program under 
evaluation and the people whose decisions will affect the future of the programs". 


Stakeholder participation has been recognized as a critical factor for bringing about 
change in developing educational performance monitoring systems (Henry, Dickey, and Areson, 
1991); systemic changes in a rural school district (Self,1994); and participation in evaluation 
designs of educational programs (Greene, 1987). Bradley and Earl (1992) and Garaway (1995) 
argued, however, that participatory evaluation models differ from the stakeholder-based model 
on several grounds including the number of stakeholders, the role of the stakeholders, and the 
function of the evaluator. Garaway (1995) suggested that the participatory model involves a 
relative small number of primary users. Greene (1987) suggested that criteria for choosing 
stakeholders should consider not only potential for making use of the findings, but also having 
some authority (administrators, program staff) to make dcisions. Greene also indicated that 
"legitimate program stakeholders are those considered to be directly involved with the project 
and both adversely and beneficially affected by the project." 


Role of the Evaluator 

Changes in the role of the evaluator has also been associated with the participatory 
approach. The traditional remoteness and lack of involvement of the evaluator has been 
criticized in the literature (Worthen and Sanders (1987). In the participatory approach, the 
evaluator assumes different roles. MacDonald (1977) sees the evaluator as “broker in exchanges 
of information between groups". Fetterman (1993), spoke of “evaluators as coaches or 
facilitators to help others conduct their evaluation®. Garaway (1995, p. 87), suggested that the 
role of the evaluator” goes beyond being the principal investigator and participant observer to 
becoming a facilitator." She added that “as a facilitator, he/she becomes a learner, arbitrator, 
and teacher, developing local skills and promoting an interactive learning environment". 
Cousins & Earl (1992) maintained that the evaluator is responsible for technical support, 
training, and quality control and that these are essential components for the successful 
completion of the project. 


Empowerment of Participants 

The empowerment rationale has received considerable support in the literature as some 
evaluators advocate using the participatory approach to empower participants associated with a 
program. Empowerment is viewed in different ways. In the developing worla, empowerment 
is seen as aimed at empowering the dominated groups so that they will be able to critically 
perceive their place within social systems (Freire, 1970; Brunner & Guzman, 1989). Another 
way to see empowerment is in terms of knowledge gained as a result of involvement in the 
program development process. The authors believe that as the participants gain more knowledge 
and skills, their decisions are better informed and, consequently, their organizations (schools) 
will become more effective. However, for empowerment to occur, certain requisites must be 


— od 


in place. First, the participants need to be motivated to participate in the program. Second, 
they need to see the benefits of their participation, and, finally, their organizations (schuols) 
need to be committed to the learning process as a route leading to improvement. 


Methodology 

Participatory evaluation usually raises questions of credibility and bias. The conduct of 
the evaluation should be guided by the appropriate professional standards set by the Joint 
Committee on Standards for Educational Evaluation (1994). The accuracy standards, in 
particular, are intended to ensure that an evaluation will provide technically sound information. 
The role of the evaluator will be crucial to ensure that those standards are met. Choosing a 
particular method is basically the same as it is for other evaluation approaches. 


Utilization 

Perhaps the most powerful definition of participatory approaches comes from the 
literature on utilization, Since the seventies, participation of stakeholders in management and 
evaluation has been seen as improving the responsiveness of the process and the utilization of 
evaluation findings. Several models were proposed as an answer to traditional approaches (e.g., 
Stake, 1985; Parlett & Hamilton, 1977). The common premise among these models was that 
meaningful participation of individuals who have a vested interest in the program to be evaluated 
will enhance utilization. The authors contend that utilization should be considered broadly to 
include use of the results after the study is conducted. The engagement in the process during 
the conduct of the study also constitutes use, especially when stakeholders learn roles which 
benefit future evaluator. 


THE CHECKLIST APPLIED TO THREE CASE STUDIES 
OF PARTICIPATORY APPROACHES 


The "Evaluation Checklist for Critiquing Participatory Planning & Evaluation Projects" 
(Appendix A) is used in this section to briefly examine participatory approaches used in three 
management and evaluation projects conducted by the Program Monitoring and Evaluation unit 
in the Leon County School District. Strengths and weaknesses of the projects are described for 
each of the six areas referred to in the checklist. The overall value of the particular 
participatory application is addressed according to those criteria. 


' Project #1 
A High Stakes Participatory Evaluation of an Innovative School Program 


Green and Morris (1994) provided the following brief description of the purpose, setting, 
and overail conduct of the high stakes participatory evaluation project treated here. 


"This project focused on the status and the ultimate decisions regarding 
modifying and continuing an innovative middle school education program...The 
project efforts and processes were used as indicators for evaluating the approach." 


"An evaluation of this primarily self-contained program was called for by 
the superintendent in September, 1993, in order to help resolve continuing 
disagreements between stakeholders with bipolar positions regarding both the 
conduct and outcomes of the program...[T]he school principal, on behalf of the 
site decision-making council, requested that the district evaluation unit oversee the 
evaluation. The evaluator assigned to the project met with the site council and 
facilitated planning a general course of action from alternative evaluation designs. 
Consequently, the site council accepted a general design, appointed a special 
evaluation task force composed of volunteer teachers and a member of the 
program faculty, and invited the evaluator to be part of the evaluation team." 


Regarding the outcomes and impor ance of the project, Green and Morris (1994) wrote 
the following synopsis. 


"The results of the evaluation triangulated with and documented what was 
known historically...about the feelings and thoughts of the stakeholder groups. 
The overwhelming majority of the program students and parents wanted the 
program to continue with little modification. However, the faculty and general 
faculty/staff were overwhelmingly opposed to certain characteristics of the 
program...As a consequence, the new administration of the school, in conjunction 
with interested teachers and site council members and district office staff, 


envisioned, developed, and initiated two new innovative educational programs for 
the school. The new programs incorpoorated characteristics the faculty survey 
results indicated were important...[T]he superintendent, and district personnel, 
and willing administrators of another school met with the parents and envisioned, 
developed, and initiated another innovative educational program in the second 
school." 


On the surface, this evaluation project could easily be mistaken as a process which 
facilitated arrival at a political decision. Indeed, it was born of disparate feelings between major 
stakeholder groups and produced little information beyond documenting the positions and 
perceptions of the stakeholders. (Given the short timelines and particular intrinsic barriers, 
achievement and other outcome information proved to be of little value in judging the worth of 
the program.) However, the worth of the evaluation process proved to be in understanding and 
reporting the faculty's and administration's positions. Their positions were useful in 
conceptualizing and developing new programs which fitted the stated overall objectives of the 
school. The fact that the process was participatory (the relevant stakeholders enjoyed significant 
status and provided some input) contributed to program parents and students appealing the site 
council’s final decisions and participating in building a new program for themselves and others 
of similar mind. It was the role of the facilitauing evaluator to help the site decision-making 
council (which by Florida law was responsible for resolving the issue) understand professional 
evaluation standards and implement quality methodology. The following is a brief discussion 
of the project according to the six areas treated in the evaluation checklist. A letter grade and 
justification are provided for each checklist question. 


By current Florida law, the process of managing and improving schools must be 
participatory in that representative councils composed of teachers, parents, students, and 
community must be actively represented in decision-making bodies. In this evaluation project, 
relevant stakeholders were represented by the site council, and the site council commissioned 
the evaluation and approved the evaluation design and conduct. However, the council went a 
step further by requiring that the evaluation include the evaluative judgements of the whole 
faculty, the program parents, and program students. (Due to limited time and resources as well 
as the belief that non-program parents and students would not have enough knowledge or interest 
to make valid judgements, potential program parents and students were represented by the 
faculty acting as surrogates.) The council created a task force of teachers and the evaluator to 
be responsible for carrying out the evaluation and reporting back to the council. (It is interesting 
that the task force included no parents or students.) The relevant checklist questions are 
addressed below. 


C "Fair" 1, The initial evaluation questions were determined by the 
stakeholding groups? 

C “Fair® 2. The initial goals of the project were redefined as a result of 
stakeholders’ input? 


10 


A "Excellent" 3. The final project goals focused on providing information or 
products for use by decision-makers in taking appropriate actions? 


B "Good" 4, The relevant stakeholders were identified appropriately through 
negotiation with interested groups? 


C "Fair" 5-7. The relevant (agents, beneficiaries, victims) were adequately 
represented in the working group? 


Participation of stakeholder groups in determining the initial evaluation questions and 
refining the questions was graded as "Fair" because while a legal representative body determined 
the goals and operationalized the questions, no attempt was made to engage relevant stakeholder 
groups. The initial evaluation goals were determined by the legal representative body, the site 
council. Its task force was directed to ask the relevant stakeholders to judge the worth of the 
project and identify their affective sentiments as well as determine, analyze, and report back 
relevant achievement and management factors of the program. A task force chairperson was 
selected by the site council, but membership was open—program faculty could and did join the 
task force, Parents were not solicited, and task force members lamented this oversight as the 
project progressed. The evaluator facilitated the task force in operationalizing the mission 
statement of the site council and in operationalizing the final questions. The operational 
questions for the various data collection instruments were fieldtested with similar stakeholder 
populations. However, relevant stakeholders did not directly participate in refining the 
evaluation questions. 


The focus of the evaluation was "Excellent"—the endeavor to obtain evaluative 
information for the decision-makers was legitimate, and work proceeded in a purposeful manner 
throughout the project. The weight of the evaluation responsibility permeated the whole school 
and was especially felt by the site council, the task force, and the faculty. The evaluator felt 
that the perceived seriousness added significantly to the validity of the results. 


Because most of the relevant stakeholder groups were appropriately identified during the 
negotiation with the site council, the authors felt that a grade of "Good" was justified for this 
question, Excellence was not achieved because potential program parents and students were not 
identified as a unique stakeholder group—the school faculty did represent them in a surrogate 
role. Primary reasons for not treating them as a unique set of stakeholders included the time 
and resources which would have been required to identify and solicit their input. 


While some agents, beneficiaries, and victims participated meaningfully in the working 
groups, the participatory nature of the evaluation process may have been more meaningful had 
the groups included victimized program students and parents—the authors were not sure that the 
value of the process would have actualy increased significantly had they been included. Because 
these stakeholders were not included, a grade of “Fair® was assigned here. 
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The position of the evaluator was one of the most intriguing characteristics of this study. 
The evaluator established early in the process that his job was to facilitate the evaluation but he 
would not be responsible for either the quality of the transactions or the outcomes. As a 
facilitator, he adopted various temporary roles including facilitator of meetings, resource, 
teacher, coach, and worker to help the task force and the site council to maximize the quality 
and efficiency of its procedures and products, This allowed him to develop adequate working 
relationships between the various stakeholder groups. The bipolarity of the general attitudes and 
sensitivity about the program required him to be vigilant of his role and also restricted the 
information flow to various stakeholder groups. The evaluator and the task force never found 
a truly acceptable way to provide adequate information to program parents regarding goals, 
products, and decisions. In part, because of this, the parents appealed the consequential 
decisions of the site council. 


A "Excellent" 8. Trust was developed and established between stakeholders and 
the evaluator? 


A "Excellent" 9. The evaluator was a facilitator and mediator for the working 
group? 

D “Poor" 10. The represented stakeholders were made aware of the goals 
of the project? 

D "Poor" 11, The represented stakeholders were made aware of products 


and decisions where appropriate? 


For the reasons provided below by Green and Morris (1994), the relationships and roles 
of the evaluator were graded as "Excellent" while communication with the stakeholders were 
rated as "Poor." 


"...a strength of this project was the trusting relationship which developed among 
the task force members and the evaluator. The evaluator and the task force 
clarified his role in the initial phases of the project--he would be a facilitator 
whose roles would be mostly determined by the barriers encounted by the project. 
He would advise and offer strong recommendations as the work progressed. He 
would not be responsible for the evaluation results, only for helping the task force 
complete the evaluation and report the results to the site council. Because the 
rules were observed, the various subgroups sought his advice and council 
repeatedly. Furthermore, true to the spirit of the initial contract, he was 
proactive in completing tasks independently and providing temporary leadership 
where necessary." 


A weakness of the project was that the two stakeholder groups most 
affected by the project (program students and parents) were not directly informed 
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of the goals and true nature of the evaluation until the parents program support 
group requested the task force chairperson to attend one of its meetings. (These 
two groups did receive some information through the program teacher who served 
as their agent on the task force.)" 


The methodological activities which were planned and accomplished are described in 
detail by Green and Morris (1994) and are not discussed here. The conclusions reached by these 
authors were that while the methods and reports were plagued with problems one might associate 
with persons other than professional evaluators, and while stakeholder representation was limited 
in the reporting process, the resulting information was meaningful to the decision-makers. Great 
effort and enthusiasm was expended by the task force. Triangulated data obviated few 
significant threats to internal validity, and the data were used to conceptualize and implement 
new programs which agreed with the goals and objectives of the school. Apparently, one of the 
greatest disappointments was that a formal written report was not delivered to the site council 
because of limited time and resources—substantial and quality information was reported in pieces 
to the site council. This points out one of the most important characteristics of internal 
evaluation at the school level, all phases must fit within the limited resources of the schooi. 
Those phases which require the most time and effort of individual teachers and administrators 
must be streamlined to functional processes. In this project, a basic reporting method was 
chosen which efficiently utlized teachers’ time, an oral presentation with data handouts. 
Program recommendations were delivered in a letter from the task force chairperson addressed 
to the site council. 


C "Fair" 12. The methods yielded information and products which met 
professional standards? 

B "Good" 13. The methods yielded information and products which were 
appropriate for the goals? 

B "Good" 14, The data collection instruments/products were developed 

through a collaborative process which meaningully involved 

stakeholders? 

C "Fair’ 15. The results and products were communicated or delivered 
effectively to decision-makers? 

F "No" 16. The represented stakeholders were meaningfully engaged in 
reporting the results? 


Probably the most intriguing characteristic of this evaluation was that it was used to make 
a radical decision about the future of a particular class of projects in the school. Not only was 
an existing program terminated, but new programs were conceptualized and implemented based 
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on the results of the evaluation of the former program. Altogether the resulting information and 
participation in the process allowed almost all of the relevant stakeholder groups to become 
empowered and engaged in structuring future programs. It is significant that many of the task 
force teachers played significant roles in developing, implementing, and managing the new 
programs and have been engaged in follow-through evaluations of the new programs during the 
1994-95 school year—thereby, maintaining and building a cadre of “program evaluators” with 
some experience. The authors’ grades for the relevant checklist questions appear below—no 
grade was provided for the final question because follow-through evaluation for 1994-95 has not 


been completed. 
A "Excellent" 17. The evaluation information and products were used effectively 
by decision-makers to take appropriate actions? 
A "Excellent" 18. The relevant stakeholder groups became more empowered as 
a result of the project? 
"Don Know" 19. Effective follow-through evaluation of consequential actions 


occurred? 


Project #2 
Conflict Mediation Tracking System (CMTS): A Self-Evaluation 
Model for School Programs 


This project was conceived to help schools with conflict mediation programs effectively 
manage and evaluate their programs by developing a system to collect and maintain their data. 
The need for this project resulted from the Florida Blueprint 2000 Accountability Law. This 
legislation mandated that schools prepare individual plans for improving safety and decreasing 
violent incidents on their campus. The Program Monitoring and Evaluation unit received a grant 
to develop a data system for conflict mediation programs in the schools which would help them 
respond to the mandate. 


The CMTS was developed using a participatory approach to planning and development. 
One of the first actions of the project was to organize a group composed of school personnel 
experienced in conflict mediation programs, subject matter experts, a university consultant, and 
district technical support staff. The group devised the goals and objectives of the project, the 
work procedures, and developed the system. The evaluators managed the planning details, 
secured funds, collected literature, facilitated the formulation of the goals of the project, ensured 
the professional quality, drafted the instruments according the group’s guidelines and helped the 
participants to reflect upon the nature of their projects. As a result of the collaboration, the 
CMTS was developed by July of 1994, and was implemented in voluntary schools during the 
Fall of 1994, 


The presence of a university consultant who acted as a subject-matter expert and lended 
credibility was a unique feature. He was also useful in acting as a broker between district and 
school staffs; thereby, avoiding conflicts and increasing the effectiveness of the approach. (The 
change from a district-initiating to a school-initiating approach was a prominent issue among 
stakeholders at that time.) 


It should be noted that this project was not an evaluation study, therefore, the "nuts and 
bolts" of problem formulation, evaluation questions, instrument selection, data collection, 
analysis, interpretations, recommendations and reporting were not dealt with by the participants. 


The authors believed that the fact that the process was participatory contributed to school 
ownership of the products and increased utilization. The following is a brief discussion of the 
project according to the six component areas of the evaluation checklist. 


The stakeholders in this project included site decision-makers such as teachers, guidance 
counselors, evaluators, technicians, and the un.versity consultant. The number of active 
participants was approximately twelve (school staff members constituted approximately half of 
the membership). The primary users (school staff) were selected by invitation. The main 
selectrion criteria were knowledge of and commitment to conflict mediation programs in their 
schools. Random selection of stakeholder is sometimes recommended in the literature, but, in 


this case, the nomination process had some advantages. By including only a few committed site 
personnel, balancing and resolving various interests, opinions, and values were not a problem. 
On the other hand, the desire to keep the group small and manageable kept some beneficiaries 
(including students) out of the process. 


Although the initial questions for this project were not defined by the stakeholders, they 
were involved in all phases of the project as soon as the grant was approved. They were 
instrumental in redefining the goals so that the goals were consistent with schools’ firsthand 
knowledge about program features. This early involvement and ownership probably increased 
the utilization of the products in their own schools and affected acceptance by other schools 
which were not directly represented in the group. Such features are crucial to participatory 
approaches. Therefore, “Excellent* and "Good" ratings were given to these components. 


C "Fair" .1, The initial evaluation questions were determined by the 
stakeholding groups? 

A "Excellent" 2. The initial goals of the project were redefined as a 
result of the stakeholders’ inpus? 

A "Excellent" 3. The final project goals focused on providing information 
or products for use by decision-makers in taking 
appropriate actions? 

B "Good" 4. The relevant stakeholders were identified appropriately 


through negotiation with interested groups? 


B "Good" 5. The relevant (agents, beneficiaries, victims) were 
adequately represented in the working group? 


A very high degree of trust was established in this project. This probably ocurred 
because the evaluators assumed responsibility for technical support, training, and quality 
control, but conducting the project was a joint responsibility. The group was engaged in the 
instrument design, formating, and dissemination of results. Also, the presence of the university 
consultant (whom the group was professionally familiar with) reinforced the credibility of the 


evaluators. ks 
A “Excellent® 8 Trust was developed and established between 
stakeholders and evaluator(s)? 
A "Excellent" 9. The evaluator(s) were facilitators and mediators for the 
working group? 
A "Excellent* 10. The represented stakeholders were made aware of the 
goals of the project? 
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A “Excellen.° 11. The represented stakeholders were made aware of 
products and decisions were appropriate? 


Issues related to validity and accuracy of the findings were not dealt with in this study. 
However, the instruments developed proved to be of high quality. This was evidenced by the 
utilization of the instruments during the second year of the grant when schools were asked to 
implement the Conflict Mediation Tracking System. Furthermore, in the process of developing 
the evaluation model and designing the instruments, the evaluator performed a pedagogical role 
in teaching the basic skills of systematic inquiry and instrument development. 


NR “Not relevant" 12. The methods yielded information and products which 
met professional standards? 

A "Excellent® 13. The methods yelded information and products which 
were appropriate for the goals? 

A "Excellent" 14. The data collection instruments/products were 


developed through a_ collaborative process which 
meaningfully involved stakeholder — 


In a brief conversation with the participants, the authors had the opportunity to ask some 
of the participants how they felt about the project. They stated that the interaction and 
collaboration with the evaluators had a three fold impact: it allowed them to fully participate in 
the group work process; it enhanced their understanding regarding evaluation; and it gave them 
a set of material and forms to use in the evaluation of their projects. This is encouraging when 
one considers that school staff members who are likely to participate in evaluation activities do 
not have sufficient research experience and knowledge to carry out the tasks. The authors feel 
that the school staffs were "knowledge empowered” as a consequence of the project. 


The participants had the opportunity to report the results of the project with other school 
staff. A workshop was organized during the summer and the participants themselves presented 
the materials. 


While it is impossible to know the extent of the utilization of the findings and products, 
it is reasonable to conclude that the participatory approach (a) helped the stakeholders to 
articulate the data needs and outcomes of the programs (conceptual use) and (6) stimulated 
action (products developed). The final proof of success will be demonstrated in the report of 
the second year grant. 


A "Excellent" 15. The results and products were communicated or 
delivered effectively to decision makers? 
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"Excellent® 


"Good" 


"Good J 


"Excellent" 


16. The represented stakeholders were meaningfully 
engaged in reporting the results? 


17, The evaluation information and products were used 
effectively by decision makers to take appropriate actions? 


18. The relevant stakeholder groups became more 
empowered as a result of the project? 


19. Effective follow-through evaluations of consequential 
actions occurred? 
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Project #3 
Technological Initiatives Guide: Leon County Pilot Schools Share Their Experiences 


The document entitled Technological Initiatives Guide: Leon County Pilot Schools Share 
Their Experiences—winner of the 1994 AERA, Division H, innovative reporting award—was 
a post hoc summary of a participatory evaluation as well as a mechanism by which 
technologically-endowed schools could share their technology initiatives with other schools. 
In essence, eight schools shared large bond funds to acquire technology and implement 
classroom and management applications in 1989-90. They were required to share their 
experiences among themselves and with schools which did not receive the funds. In 1993, the 
district inquired as to what technological initiatives had been taken by these schools which were 
worth sharing with other schools. The program evaluation unit was asked to devise a post hoc 
instrument whereby schools would evaluate their own progress and report their most worthy 
initiatives in a manner that would allow them to pass their evaluations of their best applications 
to those in the district who might attempt to replicate or improve them. The resulting document 
contained a history and summary of the bond-funded program as well as single-page entries for 
each worthy “technological initiative’ identified by the eight schools (see Green, 1993). The 
initiative pages were designed by a graphic design company so that users could quickly scan the 
salient attributes of the initiatives. In order to arrive at this information, schools were asked to 
evaluate their best intiatives in a survey. They described the characteristics of the the initiatives, 
appropriate environments for use, requisite expertise, hardware and software (requisites and 
desired), training prerequisites and strategies, special attributes and recommendations, and their 
overall evaluations of quality, helpfulness, friendliness, and robustness. Their responses to the 
survey were then entered in the single page layouts. 


The authors are intrigued by the following paradox. While the guide received some 
professional acclaim, informal follow-ups in the district indicated that it was used very little by 
the intended primary audience (teachers and school administrators). The following critique by 
a principal investigator of the project offers some insight as to why this occurred. 


The characteristics of the document were designed specifically for easy and meaningful 
use by school decision-makers. Special care was taken to offer them a highly readable and valid 
set of experiences which would help them in making tough, technological decisions. The 
technical qualities and intent of the main product were validated when it received some national 
acclaim. Therefore, the project goals and products were rated as "Excellent." However, 
informal investigations of the actual use of the document by decision-makers evidenced that little 
use has been made of the document. It is probably legitimate to attribute this to the absence of 
primary beneficiaries (teachers and school administrators) in the initial planning stages. ("Poor* 
ratings were assigned below to these attributes.) Involvement of the project schools was only 
negotiated when interviews were requested and surveys were distributed to collect the necessary 
information. The beneficiaries of the information (other schools to whom the documents were 


19 


to be distributed) were neither involved in the project nor officially informed of the project until 
. the documents were distributed. They were never consulted as to their needs or desires for 
information—therefore, a grade of "F* was given for questions 5-7. Because formal follow- 
through evaluation did not occur, it was not confirmed that the document would or did fulfill 
the technological planning goal. (However, much effort was expended in logically deducing 
schools’ needs and in planning and implementing a process which would require very little of 
schools’ time and resources—time was considered a scarce school resource.) 


D "Poor" 1, The initial evaluation questions were determined by the 
stakeholding groups? 
D "Poor* _ 2. The initial goals of the project were redefined as a result of 


stakeholders’ input? 


A "Excellent" 3. The final project goals focused on providing information 
or products for use by decision-makers in taking 
appropriate actions? 


C ‘Fair + 4, The relevant stakeholders were identified appropriately through 
negotiation with interested groups? 


F "No" 5-7. The relevant (agents, beneficiaries, victims) were adequately 
represented in the working group? 


It was important for the project to be non-threatening in order to encourage schools to 
make efforts to evaluate their own initiatives. Therefore, administrators vere interviewed 
regarding the willingness and ability of their school to participate. It was important during this 
process that the evaluators develop a trusting relationship with the administrators and show them 
how the products might benefit other schools as well as demonstrate schools’ effective use of 
the bond money. In the same vein, the evaluators felt that developing the project as a self- 
evaluation would be less intimidating and would gain more useful information than an external 
evaluation. Furthermore, schools would be able to take pride in evaluating and demonstrating 
their own effective accomplishments and would be given the opportunity to professionally advise 
others as well as have their names in a widely-distributed, professional document. In other 
words, the project was constructed to be professionally appealing and to make a decided 
contribution. With the district providing services in reviewing and improving schools’ 
initiatives, the evaluation developed a collaborative feel. Review of the initiative pages 
demonstrates that the teachers and adminstrators took their roles seriously in providing good 
advice to other professionals. Contrasted with the early stages where relevant agents and 
beneficiaries were not consulted about the goals and evaluation questions, the providing 
stakeholders were treated as important team members. 


As a result of the milieu described above, the organization and negotiation process was 
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graded above average with the highest grade given for the trusting relationship between the 
evaluators and the stakeholders. The most outstanding characteristics of this project were the 
quality of its ultimate product and the trusting relationships which were developed to assure 
its quality. 


A "Excellent" 8. Trust was developed and established between stakeholders and 
the evaluator? 


B "Good" 9. The evaluator was a facilitator and mediator for the working 
group? 


B "Good" 10. The represented stakeholders were made aware of the goals 
of the project? 


B "Good" 11. The represented stakeholders were made aware of products 
and decisions where appropriate? 


In this section, the authors will not dwell on the qualities of the resulting Technological 
Initiatives Guide. It seems more important to focus on some of the project characteristics which 
fated the guide to primarily reside on the dusty top shelves of schools’ book cases. The primary 
goals of the project were to document both the most advanced technological 
accomplishments/experiences of the bond money schools and to distribute their most useful 
evaluative information to their less fortunate peers. The first goal was accomplished, therefore, 
the project was not a failure. However, the second goal depended on how useful the evaluations 
were to the target population at the particular point in time. The technology demonstrated most 
often in the guide was late eighties and early nineties, and the beneficiaries were mostly 
interested in middle nineties technology because all Leon County schools had acquired at least 
some early nineties technology—as was evidenced in post hoc informal queries and through 
general enlightenment of the evaluators. No formal evidence exists to confirm the guide's actual 
use by either school or district level decision-makers regarding purchases, new applications, or 
planning for technology. Had the beneficiaries been included in the development of the goals 
and the implementation of the project, the potential utilitarian characteristics of the product might 
have been clarified early in the project. As the grades below reflect, although the methods 
yielded information and products which met professional standards, meaningfully engaged at 
least some stakeholders, and were effectively communicated and delivered; they were not 
entirely appropriate for the project goals. In light of informal, opportunistic inquiries by the 
evaluators, it is assumed that the Technological Initiatives Guide, although acclaimed as 
excellent, performed poorly in stimulating appropriate actions by administrators and in 
empowering potential beneficiaries. 


A °Excellent® 12, The methods yielded information and products 
which met professional standards? 


C "Fair® 13. The methods yielded information and products 
which were appropriate for the goals? 


C "Fair" 14. The data collection instruments/products were 
developed through a collaborative process which 
meaningully involved stakeholders? 


A "Excellent* 15. The results and products were communicated or delivered 
effectively to decision-makers? 

B "Good" 16. The represented stakeholders were meaningfully 
engaged in reporting the results? 

D "Poor" 17. The evaluation information and products were 
used effectively -by decision-makers to take 
appropriate actions? 

D "Poor" 18, The relevant stakeholder groups became more empowered as 
@ result of the project? 

D "Poor" 19. Effective follow-through evaluation of consequential actions 


occurred? 


CONCLUSIONS AND LESSONS LEARNED 


Under present mandates, many school district program evaluators—most assuredly in 
Florida—find that they are bound to a new milieu where meaningful participation and 
empowerment of host stakeholders is assumed. Indeed, having been held accountable, school 
decision-making bodies have been granted both the purview of decision-making and the purview 
of internal evaluation. District staffs exist to facilitate schools’ management and accountability 
efforts. District program evaluation units, if not directed to be auditors fer boards and 
superitendents, find themselves as participants in the milieu who have much professional 
expertise to offer, but no special authority other than professional and moral for assuming 
prominent roles. In the "site decision-making" paradigm, judgements of worth must proceed 
with or without professional, internal program evaluators participating. The authors believe that 
internal evaluators can participate in this paradigm and facilitate real stakeholder empowerment 
which includes the understanding, willingness, ability, and the actual achievement of professional 
evaluation standards. Furthermore, the paradigm offers the entusiastic evaluator almost 
unlimited opportunities for facilitating evaluation. Program evaluators are challenged to assume 
roles which promote achievement of evaluation standards as well as stakeholder empowerment. 


The authors believe there is much for the internal evaluators who function in the new 
paradigm to learn and apply from the literature of participatory evaluation. The body of this 
paper contains many of the roles and activities that the authors have been experimenting with 
during the past several years as well as a checklist devised by the authors to enhance their 
abilities to critique the characteristics of program management and evaluation in the participatory 
milieu, Several lessons are listed below which have been learned in their first years of applying 
participatory approaches to school district phenomenon. 


e It seems imperative that evaluators should establish the facilitative nature of their roles 
with stakeholders during their first project encounter with stakeholders. In both the 
conflict mediation and high stakes evaluation projects discussed in this paper, the 
evaluators established their role during the first encounter, and the stakeholder groups 
began to assume some ownership of the project, immediately. Both of these projects 
were characterized by stakeholder ownership of the products and utilization of the 
resulting products. It is also noteworthy that in both projects, the informal contracts 
between the evaluators and the stakeholders allowed enough lattitude for either to assume 
responsibility for new tasks as the necessity for them became apparent. The fact that 
neither project experienced conflicts between the evaluator and the stakeholders was 
evidence that trusting relationships developed early in the projects. 


@ — Stakeholders should be included in the development of both the goals and the plans for 
conducting work because (a) they bring special knowledge and experiences to the project 
which may be invaluable in determining existing needs and (0) their interest in 
developing and utilizing products are increased if they have ownership throughout the 
project. The consequence of not including them can be a highly professional but 


minimally utilized product such as the Technological Initiatives Guide discussed in the 
main body of the paper. 


In participatory evaluation, evaluators facilitate in a complex environment which is 
fraught with politics and unexpected barriers and outcomes. Therefore, it is prudent to 
use triangulation in establishing information. In the high stakes evaluation project 
discussed in this paper, stakeholders were convinced of the worth of the final data by 
how well it seemed to correlate with other information. Stakeholders who are not 
familiar with disciplined inquiry may need the validity of the findings to be demonstrated 
to them in light of other evidence. Similar to the legal system, jurors (stakeholders) may 
make their judgements about transactions and outcomes according to the “weight of the 
evidence" rather than according to good inquiry methods which they may not be familiar 
with, 


The delivery mediums of reports may vary when stakeholders are engaged in the 
reporting process. It is unlikely that many stakeholders will have the experience, time, 
or desire to produce a written report of the quality expected of professional evaluators. 
The consequences of preparing a professional written report may be that the evaluator 
would write it and the stakeholders would not take ownership. The medium should be 
chosen to fit the abilities and qualities of the stakeholders and the environment. In the 
high stakes evaluation referenced here, the faculty stakeholders chose to prepare an oral 
report to the decision-makers supplemented by information handouts. (it should also be 
noted the faculty stakeholders later lamented that they did not prepare an inclusive 
written report. The task force chairperson felt "incomplete" until she put her own report 
and recommendations in a letter to the decision-makers. 


Stakeholder working groups may prefer to be small. It is significant to note that in both 
projects where a stakeholder working group was formed, the groups voted early in the 
process to limit the working group to less than approximately fifteen individuals for the 
same reasons, First, they felt it would be easier to work with just a few committed 
people. Second they felt they could fairly represent other professionals. 


Trust between stakeholder groups can be enhanced by a trusted figure who is responsible 
for mediating the opening contracts with the stakeholder groups. In the conflict 
mediation project, this role was performed by a university professor who was known by 
the working group members. In the high stakes evaluation project, this was performed 
by the evaluator whose first task was to help the stakeholders find ways to conduct the 
evaluation which had been mandated to help them resolve the conflict between them. 


In district internal evaluations governed the new paradigm, it is worthwhile for evaluators 
to spend time in a one-on-one pedagogical role. In education, there are many literate, 
well-meaning professionals who will take responsibility and ownership in participatory 
projects. Many of these people have pieces of the information needed to perform good 
evaluation. They just need help in rounding out their knowledge, and the facilitating 
evaluator cannot become familiar with their individual needs without working with them 
one-on-one. In the high stakes evaluation project, it was especially interesting that the 
working group included five classroom teachers with doctoral degrees—all of whom had 
some disciplined inquiry courses in the distant past. The evaluator felt that all of these 
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individuals, as well as others in the group, could tecome practicing evaluators given the 
desire and some on-the-job training. Under the site decision-making paradigm, 
individuals such as these must provide leadership, and internal evaluators should find 
ways to help them. 


Follow-through evaluation seems to be naturally enhanced by participatory approaches 
when stakeholders are engaged and committed. Educational decision-makers naturally 
feel more necessity to sponsor follow-through evaluation when their programs and 
products resulted from participatory management and evaluation approaches which were 
engaged in by their stakeholders. 


In conclusion, the most important lesson the authors have learned from their experiences 
is that participatory approaches facilitated by professional evaluators can lead to 
legitimate action by decision-makers. Furthermore, such action seems to be enhanced 
by the team spirit which is an inherent characteristic of successful participatory 
evaluation. The facilitating evaluator can become a part of the team, and evaluation 
becomes less negative. 
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Appendix A 


Evaluation Checklist for Critiquing Participatory Planning & Evaluation Projects 


Grade {A=Excellent B=Good C=Fair D=Poor F=Did 
Not Occur DK=Don't Know NR=Not Relevant} 


Project: 


Focusing the Evaluation 


1 The initial questions were determined by the stakeholding groups? 
2 The initial goals of the project were redefined as a result of stakeholders’ input? 
3 The final project goals focused on providing information or products for use by 
decisionsmakers in taking appropriate actions? 


Comments: 


Identifying Stakeholders 


4 The relevant stakeholders were identified appropriately through negotiation with 
interested groups? 
The relevant __ were adequately represented in the working group? 
5 agents {district & school staffs} 
6 beneficiaries 


7 victims 


Comments: 


Organization and Negotiation Process 


8 Trust was developed and established between stakeholders and evaluator(s)? 

9 The evaluator(s) were facilitators and mediators for the working group? 

0__ The represented stakeholders were made aware of the goals of the project? 

1___ The represented stakeholders were made aware of products and decisions where 
appropriate? 


"P| 


Comments: 


Methodology 


12____ The methods yielded information and products which met professional standards? 

13____ The methods yielded information and products which were appropriate for the 
goals? 

14___ The data collection instruments/products were developed through a collaborative 

process which meaningfully involved stakeholders? 


Comments: 


Reporting 


1$___ The results and products were communicated or delivered effectively to 
decision-makers? 
16___ The represented stakeholders were meaningfully engaged in reporting the results? 


Comments: 


Utilization & Benefits 


17___ The evaluation information and products were used effectively by decision-makers 
to take appropriate actions? 

18___ The relevant stakeholder groups became more empowered as a result of the 
project? 

19___ Effective follow-through evaluation of consequential actions occurred? 


Comments: 
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